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Q: If we encode symbols from the ensemble 
Ax = {a,b,c,d} 
Px = {1/2,1/4, 1/8, 1/8} 
using the symbol code 
С = {0, 10, 110, 111}, 


what is the probability ‘ру’ 
that a bit plucked at random from the encoded stream 
is a 1? 


A p< 
В p= 
Сор» 


vie fole юе 


Last time - symbol codes 


The ideal codelengths l are the information contents 
1 
lí = log — 
N s Pi 


The optimal symbol code’s expected length L satisfies 
H(X)<L<H(X)+1 


Өроеѕ that wrap ир сотргеѕѕіоп? 
Optimal symbol codes get within one bit per character of Shannon limit 


@dentical twins can help make compressors 


Arithmetic coding 


В ЕЯ 


Arithmetic coding 


achieves 


1 
I(x) € logs Р(х) +2 


where x is the whole file z1£2... £N. 


Data compression 


(ФАп example predictor 
P(x: | 212223...21-1) 


.. I'm afraid I don't know one,' said 
Harriet, rather alarmed at the pro 


Data compression 


@An example predictor 
P(x: | 212223...21-1) 


.. I'm afraid I don't know one,' said 
Harriet, rather alarmed at the_pro 


PPM - 'prediction by partial match’ 


to her to Бе secure of а comfortable_provision, 

and be_proved in the right, when so many people said 
was an irresistible_proof of his great good sense, 
"Mrs. Bates, let me_propose your venturing on one of 


was so great a personage in Highbury, that the_prospect 


Predictions proportional to frequencies (in this context) 


Six-gram model 


Data compression 


@An example predictor 
Ра: | 122235... 241) 


“I'm afraid I don't know one,' said 
Harriet, rather alarmed at the_pro 

e_pro 

-рго 

рго 

го 
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Other uses for arithmetic coding 


Dasher 


ФЕ сеп writing Information-Efficient Text Entry 
Compression: 
Text — Bit string 
(preferably short) 
Writing: 


Text «— Gesture 
(preferably brief) 


Dasher B 


www.inference.phy.cam.ac.uk/dasher/ 


Other uses for arithmetic coding 


Dasher 


ФЕ сеп writing Information-Efficient Text Entry 
Compression: 
Text — Bit string 
(preferably short) 
Writing: 


Text «— Gesture 
(preferably brief) 


@Efficient generation of random samples 


ays Хэний 
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Arithmetic coding 


В ЕЯ 


N ВЕК СОКОМ 
including the 
DIEHARD BATTERY OF TE 
OF RANDOMNESS 


и 
FORNDATION 
Grants 


DMO 
DMS YT 


4 - % - 2 094. G [е мириса fsu.edu/jpub/diehard/cdrom/ 
THE MARSAGLIA 
RANDOM NUMBER CDROM 
including the 
DIEHARD BATTERY OF TESTS 
OF RANDOMNESS 
Reseach 
a LET 
тненатена. a 
patsy 
коб он SUPERCOMPUTER 2 
рм уу, ณ แต่ 09-Dec-1995 10:21 9.5M 
DNS 920472 INSTITUTE 09-Dec-1995 10:23 9.5M 
09-Dec-1995 10:249.5M 
Ld เง 09-Dec-1995 10:26 9.5M 
ss George Marsaglia 
м ре тт 09-Dec-1995 10:27 9.5M 
09-Dec-1995 10:29 9.5M 
State 09-Dec-1995 10:30 9.5M 
UNIVERSITY 09-Dec-1995 10:329.5М 
—m . m m mM ท ร (09-Dec-1995 10:33 95М. 


The Marsaglia Random Number CD-ROM contains 5 billion random bits, divided into sixty 
10-megabyte files 


It was developed and distributed under National Science Foundation Grants DMS-8807976 and 
DMS-9206972, at Florida State University 


Three cards 


pe 
ED 
p. 


ur (0, ) 278 y рай veta 
| Ue Y >A 
— 一 一 一 


г. ETE. face це" 


1 


о 100 5 още 


Тпгее cards 


Q 
What is the probability that 
the other face is white， 
given that you are now seeing a white face? 


Communication over noisy channels 


(Noisy channels o o 
x T 
1 
1 51 


inference 1-7 
‘information measures 


ซี 6 อ เซ เซ ๑ ๒ 


The rules 


THREE Doors 
OD UN 
Р "ze 
o Host оргль о соле 
another door; к | 
promising she prie ‚тент 
Gill no be revealed . " dec 4. 


о Player chooses do stick 
OC swith e, Rawe 
Wet's behind Mia original 
doc, ос the other closed 
door. 


о 


(Player chooses door 1 


(Host opens door 3, revealing nothing, as promised 


фар 


STICK? 


A should stick 

B should switch 

C makes no difference 
Z don't know 


Information measures 


H(X,Y) 


H(X) 


но) 


HXIY) 


I(X;Y) 


H(Y|X) 


Mutual information for the BSC 


0 E 0 a- [17 f | 


i 4 


lef 
Consider f = 0.1. 


Assume input distribution Px = {po, pı} = {0.9.0.1}. 


Find 
I(X;Y) = H(Y)-H(Y|X) Im 
= Н»(0.18) — H2(0.1) 
= 0.68 — 0.47 ын อ 
= 0.21 bits. 


= ы 1 А 
Complete the curve 


Inference for channels 


Channel Q defines conditional probabilities 
Р(у|=) 


If we choose an input distribution P(z), 
we have a joint distribution 
Р(т.у) = P(x) P(y|x) 


with which we can do inference. 


во че о о > 


Information measures for noisy channels 


Channel Q defines conditional probabilities 
P(y|x) 5 
If we choose an input distribution P(x), 
we have a joint distribution 
P(x,y) = P(x) P(y|x) 
for which we can compute the mutual information 


I(X;Y) = H(X)-H(X|Y) 
H(Y) - H(Y | X) 


H(X,Y) 


H(X) 


HY) 
но) IY) | нолх) 


Mutual information for the BSC 


0 E 0 a- [17 f | 


i 4 


lef 
Consider f = 0.1. 


Assume input distribution Px = {po, pı} = {0.9.0.1}. 


Find 
I(X;Y) = H(Y)-H(Y|X) Im 
= Н»(0.18) — H2(0.1) 
= 0.68 — 0.47 ын อ 
= 0.21 bits. 


= ы 1 А 
Complete the curve 


Recommended homework 


@Reading: Chapters 1-6; Advance reading: Chapters 8, 9, 10 


@See handout 2 on website for more recommended exercises 


Huffman programs huffman.p, huffman.py are on website 
also a ‘bent coin’ file 0010000... as a compression benchmark 


